Linear Response Formula and Generalized Belief Propagation for Probabilistic Inference

In probabilistic inference for signal processing, medical diagnosis, code theory, digital communication, machine learning and so on, belief propagation (BP) is one of powerful approximate methods to calculate a belief for each node within a practical computational time[1]. Recently, it has been pointed out that the BP has some mathematical structures in common with advanced mean-field (MF) methods in the statistical-mechanics[2]. Yedidia et al.[3] have pointed out that the update rule of generalized belief propagation (GBP) is equivalent to a cluster variation method (CVM) that is one of the advanced MF methods. However, it is difficult to calculate the marginal probability in a pair of nodes, which are not connected directly to each other, by means of the BP, because the recursion formulas in the BP are constructed only from some marginal probabilities. In Refs.[4, 5], it was mentioned that the correlations in every pair of nodes in a probabilistic model can be calculated by combining the native MF approximation with a linear response (LR) theory. The present author extended the naive MF approximation in their framework to the CVM[6] for the case that each node has two states and derived the marginal probability in every pair of nodes as an inverse of matrix. On the other hand, Welling and Teh[7] constructed an update rule of the GBP to calculate pairwise marginal probabilities by using the LR theory. In this paper, we derive the general formula for the marginal probability in every pair of nodes for any probabilistic models by combining the CVM with the LR theory. for the case that each node has 1E-mail: [email protected]